Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: add option to create JSON info files #2192

Open
wants to merge 2 commits into
base: master
from

Conversation

@aparcar
Copy link
Member

commented Jul 3, 2019

build: create JSON files containing image info

The JSON info files contain details about the created firmware images
per device and are stored next to the created images.

The JSON files are stored as "$(IMAGE_PREFIX).json" and contain some
device/image meta data as well as a list of created firmware images.

An example of openwrt-ath79-generic-tplink_tl-wdr3600-v1.json

    {
      "id": "tplink_tl-wdr3600-v1",
      "image_prefix": "openwrt-ath79-generic-tplink_tl-wdr3600-v1",
      "images": [
        {
          "name": "openwrt-ath79-generic-tplink_tl-wdr3600-v1-squashfs-sysupgrade.bin",
          "sha256": "60ef977447d57ffe406f1f6170860be8043654d961933b73645850b25c6a1990",
          "type": "sysupgrade"
        },
        {
          "name": "openwrt-ath79-generic-tplink_tl-wdr3600-v1-squashfs-factory.bin",
          "sha256": "c6fae436b13f512e65ef05c0ae94308dd1cc9e20fd929dd3e0422574fe58d2b5",
          "type": "factory"
        }
      ],
      "metadata_version": 1,
      "model": "TL-WDR3600",
      "supported_devices": [
        "tplink,tl-wdr3600-v1",
        "tl-wdr4300"
      ],
      "target": "ath79/generic",
      "title": [
        "TP-Link TL-WDR3600 v1"
      ],
      "variant": "v1",
      "vendor": "TP-Link",
      "version_commit": "r10764-84c103509a",
      "version_number": "SNAPSHOT"
    }

Signed-off-by: Paul Spooren <mail@aparcar.org>


build: add JSON info merge script

Script creates three different files types with different functions:

* map.json

Stored in each target/subtarget folder containing a mapping between
$(SUPPORTED_DEVICES) and JSON image info files. With this file a router
can automatically obtain a sysuprade firmware image.

    {
      "devices": {
        "netgear,r7800": {
          "info": "openwrt-ipq806x-generic-netgear_r7800.json",
          "sha256": "9d24baeba60c4f592838bf7fdf2a1f87b61d4cd1432953d2960315a37149b89a"
        },
        "r7800": {
          "info": "openwrt-ipq806x-generic-netgear_r7800.json",
          "sha256": "9d24baeba60c4f592838bf7fdf2a1f87b61d4cd1432953d2960315a37149b89a"
        }
      },
      "metadata_version": 1,
      "target": "ipq806x/generic"
    }

This mapping style allows device identifiers to change between releases, as it
happend between ar71xx and ath79.

Sha256sums are stored to verify file integrity, later more on that.

This files is stored in $(OUTPUT_DIR) and contains a list of all
targets where images exist for.

* targets.json

    {
      "metadata_version": 1,
      "targets": {
        "ar71xx/generic": {
          "path": "targets/ath79/generic",
          "sha256": "434bb3de0e5e0a1240a714d08360667daeb11f92b36cea4fb590046392f2f3a7"
        },
        "ath79/generic": {
          "path": "targets/ath79/generic",
          "sha256": "434bb3de0e5e0a1240a714d08360667daeb11f92b36cea4fb590046392f2f3a7"
        },
        "ipq806x/generic": {
          "path": "targets/ipq806x/generic",
          "sha256": "6afba149bc0b0f22e19ebb49e10d7e2a98e80286f84eafcd13b65384f71f401f"
        },
        "x86/64": {
          "path": "targets/x86/64",
          "sha256": "0f445ca9807d7f21624fff802f33320b30247eedfca636237317a144729c02e1"
        }
      }
    }

This mapping style allows device targets to change between releases, as it
happend between ar71xx and ath79. Devices searching for ar71xx are forwarded to
compatible deivces of ath79.

The workflow would be the following:

* Devices request a (tbd) signed versions.json file from the server
* Parse the file on device and optain the folder containing targets.json
* Request targets.json and optain path to target specific map.json
* Request map.json and optain device specific info file
* Parse info file images for a sysupgrade
* Request sysupgrade and flash device

The versions.json would be signed containing sha256sums of targets.json files.
This allows a chain of trust:

    versions.json -> targes.json -> map.json -> device_id.json -> sysupgrade.bin

The versions.json file could look like the following:

    {
        "versions": [
            {
                "name": "19.07-SNAPSHOT",
                "path": "releases/19.07-SNAPSHOTS",
                "sha256": "e659ccba1cd70124138726efabd9dd7a60ee220bf8471a91082c7e05488cac19"
            },
            {
                "name": "Snapshot",
                "path": "snapshots",
                "sha256": "7c84e6140f35c70d6fe71cd1ca8e5d98d9f88a887a4550adf137d4fd2a1f0ea6"
            }
        ],
        "metadata_version": 1
    }

* overview.json

Contains a mapping between device title(s) and device info files. Having
this file greatly simpifies firmware image retreiveal. A (web) client can
search trhough the mapping and show the results to an end user.

    {
      "devices": {
        " TL-WR840N v5": "targets/ramips/mt76x8/openwrt-ramips-mt76x8-tplink_tl-wr840n-v5.json",
        "7Links PX-4885 4M": "targets/ramips/rt305x/openwrt-ramips-rt305x-7links_px-4885-4m.json",
        "7Links PX-4885 8M": "targets/ramips/rt305x/openwrt-ramips-rt305x-7links_px-4885-8m.json",
        "8devices Carambola": "targets/ramips/rt305x/openwrt-ramips-rt305x-8devices_carambola.json",
        "8devices Carambola2": "targets/ath79/generic/openwrt-ath79-generic-8dev_carambola2.json",
        ...
      },
      "metadata_version": 1
    }

Signed-off-by: Paul Spooren <mail@aparcar.org>

@aparcar aparcar force-pushed the aparcar:CREATE_JSON branch 4 times, most recently from 2a2bda1 to 40ed190 Jul 4, 2019

@aparcar aparcar changed the title [wip]build: add option to create JSON info files build: add option to create JSON info files Jul 4, 2019

@adrianschmutzler

This comment has been minimized.

Copy link
Contributor

commented Jul 4, 2019

After each build process all JSON files are merged in a single devices.json file only containing title and device ID.

Had to think twice about that: JSON files are NOT merged, but selective information about them is collected in devices.json ...

@adrianschmutzler

This comment has been minimized.

Copy link
Contributor

commented Jul 4, 2019

"8devices Carambola2": "8dev_carambola2"

Do you have a reason why you use the friendly name as key and the ID as value? I would have done it the other way around, so the ID is used as ID (although it technically won't matter if I got your use-case right)

scripts/add_json.sh Outdated Show resolved Hide resolved
@aparcar

This comment has been minimized.

Copy link
Member Author

commented Jul 5, 2019

After each build process all JSON files are merged in a single devices.json file only containing title and device ID.

Had to think twice about that: JSON files are NOT merged, but selective information about them is collected in devices.json ...

devices.json is like a map, on what devices are available.

Do you have a reason why you use the friendly name as key and the ID as value? I would have done it the other way around, so the ID is used as ID (although it technically won't matter if I got your use-case right)

Idea is to search trough the keys via a fuzzy search, once a device is found more detailed information are gathered via the <device_id>.json

@aparcar aparcar force-pushed the aparcar:CREATE_JSON branch 2 times, most recently from 86a128f to 4fa6e5f Jul 5, 2019

@adrianschmutzler

This comment has been minimized.

Copy link
Contributor

commented Jul 5, 2019

devices.json is like a map, on what devices are available.

I'm aware of that, I just wanted to advice on changing the part in the commit message/PR description where you say "merged" to make it clearer.

@adrianschmutzler

This comment has been minimized.

Copy link
Contributor

commented Jul 5, 2019

Idea is to search trough the keys via a fuzzy search, once a device is found more detailed information are gathered via the <device_id>.json

Well, I still would exchange key and value and then search through the values.

@aparcar aparcar force-pushed the aparcar:CREATE_JSON branch from 4fa6e5f to f4724c9 Jul 5, 2019

@aparcar

This comment has been minimized.

Copy link
Member Author

commented Jul 5, 2019

exchanged key and value, also clarified the commit message

@aparcar

This comment has been minimized.

Copy link
Member Author

commented Jul 7, 2019

@jow- please have a look

@aparcar aparcar force-pushed the aparcar:CREATE_JSON branch from f4724c9 to bd2277d Jul 7, 2019

@aparcar

This comment has been minimized.

Copy link
Member Author

commented Jul 7, 2019

Here is an example for resulting files https://chef.libremesh.org/download/json/

@aparcar aparcar force-pushed the aparcar:CREATE_JSON branch from bd2277d to e69c006 Jul 13, 2019

@aparcar

This comment has been minimized.

Copy link
Member Author

commented Jul 13, 2019

@blogic I moved the output to $(TOPDIR)/bin/json/. Would that work for you?

@aparcar aparcar force-pushed the aparcar:CREATE_JSON branch from e69c006 to 9edbe69 Jul 13, 2019

@lynxis

This comment has been minimized.

Copy link
Contributor

commented Jul 15, 2019

I would like to have a hash (sha256) of glinet_gl-ar150.json into the devices.json.
If we have this, we could sign devices.json and have a valid path down to the image. I see value in this, even if the images are signed itself.

I'm also missing a version field of the json structure. I mean the api/specification of the json field.

@aparcar aparcar force-pushed the aparcar:CREATE_JSON branch from 9edbe69 to 7c3023b Jul 15, 2019

@aparcar

This comment has been minimized.

Copy link
Member Author

commented Jul 15, 2019

The approach was changed, now maps.json and overview.json are ingame to solve two issues at once: vanilla firmware upgrades and firmware search! The script became to complex to use purse shell, but as Python3 becomes (is?) a buildroot dependency anyway, that shouldn't cause any problem.

If a (to be designed) release.json file is signed and contains checksums of the maps.json file, a secure chain is created as long as sha256 isn't broken.

@aparcar aparcar force-pushed the aparcar:CREATE_JSON branch 2 times, most recently from f8b7eff to 4d58d3b Jul 16, 2019

@aparcar

This comment has been minimized.

Copy link
Member Author

commented Jul 16, 2019

Now uses two Python3 scripts instead of messy shell scripts. Also a map file is created per subtarget.

@aparcar aparcar force-pushed the aparcar:CREATE_JSON branch from 4d58d3b to 30e0216 Jul 16, 2019

@aparcar

This comment has been minimized.

Copy link
Member Author

commented Jul 16, 2019

Now also handles ALT titles

include/image.mk Outdated Show resolved Hide resolved
include/image.mk Outdated Show resolved Hide resolved

@aparcar aparcar force-pushed the aparcar:CREATE_JSON branch 3 times, most recently from de85408 to 021ea8e Aug 19, 2019

sudhanshu16 added a commit to sudhanshu16/luci that referenced this pull request Aug 21, 2019
replaces attendedsysupgrade with sysupgrade
Utilises the JSON schema generated after build as suggested in:
openwrt/openwrt#2192 (comment)

Ability to apply fresh images (default packages only, from openwrt)
or to apply image with installed packages (from attendedsysupgrade server)

Signed-off-by: Sudhanshu Gautam <me@sudhanshug.com>
sudhanshu16 added a commit to sudhanshu16/luci that referenced this pull request Aug 21, 2019
replaces attendedsysupgrade with sysupgrade
Utilises the JSON schema generated after build as suggested in:
openwrt/openwrt#2192 (comment)

Ability to apply fresh images (default packages only, from openwrt)
or to apply image with installed packages (from attendedsysupgrade server)

Signed-off-by: Sudhanshu Gautam <me@sudhanshug.com>
aparcar added a commit to aparcar/openwrt-metabuilder that referenced this pull request Aug 22, 2019
Backport JSON info files patch
While unmerged, patch it here
openwrt/openwrt#2192

Signed-off-by: Paul Spooren <mail@aparcar.org>
aparcar added a commit to aparcar/openwrt-metabuilder that referenced this pull request Aug 22, 2019
Backport JSON info files patch
While unmerged, patch it here
openwrt/openwrt#2192

Signed-off-by: Paul Spooren <mail@aparcar.org>
sudhanshu16 added a commit to sudhanshu16/luci that referenced this pull request Aug 25, 2019
replaces attendedsysupgrade with sysupgrade
Utilises the JSON schema generated after build as suggested in:
openwrt/openwrt#2192 (comment)

Ability to apply fresh images (default packages only, from openwrt)
or to apply image with installed packages (from attendedsysupgrade server)

Signed-off-by: Sudhanshu Gautam <me@sudhanshug.com>
sudhanshu16 added a commit to sudhanshu16/luci that referenced this pull request Aug 25, 2019
replaces attendedsysupgrade with sysupgrade
Utilises the JSON schema generated after build as suggested in:
openwrt/openwrt#2192 (comment)

Ability to apply fresh images (default packages only, from openwrt)
or to apply image with installed packages (from attendedsysupgrade server)

Signed-off-by: Sudhanshu Gautam <me@sudhanshug.com>
sudhanshu16 added a commit to sudhanshu16/luci that referenced this pull request Aug 25, 2019
replaces attendedsysupgrade with sysupgrade
Utilises the JSON schema generated after build as suggested in:
openwrt/openwrt#2192 (comment)

Ability to apply fresh images (default packages only, from openwrt)
or to apply image with installed packages (from attendedsysupgrade server)

Signed-off-by: Sudhanshu Gautam <me@sudhanshug.com>
sudhanshu16 added a commit to sudhanshu16/luci that referenced this pull request Aug 26, 2019
replaces attendedsysupgrade with sysupgrade
Utilises the JSON schema generated after build as suggested in:
openwrt/openwrt#2192 (comment)

Ability to apply fresh images (default packages only, from openwrt)
or to apply image with installed packages (from attendedsysupgrade server)

Signed-off-by: Sudhanshu Gautam <me@sudhanshug.com>
sudhanshu16 added a commit to sudhanshu16/luci that referenced this pull request Aug 26, 2019
replaces attendedsysupgrade with sysupgrade
Utilises the JSON schema generated after build as suggested in:
openwrt/openwrt#2192 (comment)

Ability to apply fresh images (default packages only, from openwrt)
or to apply image with installed packages (from attendedsysupgrade server)

Signed-off-by: Sudhanshu Gautam <me@sudhanshug.com>
sudhanshu16 added a commit to sudhanshu16/luci that referenced this pull request Aug 26, 2019
replaces attendedsysupgrade with sysupgrade
Utilises the JSON schema generated after build as suggested in:
openwrt/openwrt#2192 (comment)

Ability to apply fresh images (default packages only, from openwrt)
or to apply image with installed packages (from attendedsysupgrade server)

Signed-off-by: Sudhanshu Gautam <me@sudhanshug.com>
sudhanshu16 added a commit to sudhanshu16/luci that referenced this pull request Aug 26, 2019
replaces attendedsysupgrade with sysupgrade
Utilises the JSON schema generated after build as suggested in:
openwrt/openwrt#2192 (comment)

Ability to apply fresh images (default packages only, from openwrt)
or to apply image with installed packages (from attendedsysupgrade server)

Signed-off-by: Sudhanshu Gautam <me@sudhanshug.com>
@aparcar

This comment has been minimized.

Copy link
Member Author

commented Aug 27, 2019

Please see the outcome of this years GSoC here. It's a firmware wizard which utilizes the JSON produced in this PR to allow users simple selection of firmware. It also supports images on demand for custom package selections, please have a look.

image

image

aparcar added a commit to aparcar/openwrt-metabuilder that referenced this pull request Sep 1, 2019
Backport JSON info files patch
While unmerged, patch it here
openwrt/openwrt#2192

Signed-off-by: Paul Spooren <mail@aparcar.org>
aparcar added a commit to aparcar/openwrt that referenced this pull request Sep 1, 2019
base-files: add sysupgrade-online script as POC
This is a proof of conecpt and requires further work, it show how a
device can find upgrades only in a secure way, without any extra
packages, especially nothing HTTPS related.

The online.sh "library" can be used for CLIs or ubus calls for Luci or
other webinterfaces to come.

Details about the JSON structure are explained in another PR[0].

To try the script, run `sysupgrade-online --search` or `--unattended`,
the former searches for a sysupgrade (only snapshots currently), the
latter searches and passes results to a download function, which then
downloaded the sysupgrade.

Ideally the sysupgrade would be verified aditionally via usign.

[0]: openwrt#2192

Signed-off-by: Paul Spooren <mail@aparcar.org>
aparcar added 2 commits Aug 18, 2019
build: create JSON files containing image info
The JSON info files contain details about the created firmware images
per device and are stored next to the created images.

The JSON files are stored as "$(IMAGE_PREFIX).json" and contain some
device/image meta data as well as a list of created firmware images.

An example of openwrt-ath79-generic-tplink_tl-wdr3600-v1.json

    {
      "id": "tplink_tl-wdr3600-v1",
      "image_prefix": "openwrt-ath79-generic-tplink_tl-wdr3600-v1",
      "images": [
	{
	  "name": "openwrt-ath79-generic-tplink_tl-wdr3600-v1-squashfs-sysupgrade.bin",
	  "sha256": "60ef977447d57ffe406f1f6170860be8043654d961933b73645850b25c6a1990",
	  "type": "sysupgrade"
	},
	{
	  "name": "openwrt-ath79-generic-tplink_tl-wdr3600-v1-squashfs-factory.bin",
	  "sha256": "c6fae436b13f512e65ef05c0ae94308dd1cc9e20fd929dd3e0422574fe58d2b5",
	  "type": "factory"
	}
      ],
      "metadata_version": 1,
      "model": "TL-WDR3600",
      "supported_devices": [
	"tplink,tl-wdr3600-v1",
	"tl-wdr4300"
      ],
      "target": "ath79/generic",
      "title": [
	"TP-Link TL-WDR3600 v1"
      ],
      "variant": "v1",
      "vendor": "TP-Link",
      "version_commit": "r10764-84c103509a",
      "version_number": "SNAPSHOT"
    }

Signed-off-by: Paul Spooren <mail@aparcar.org>
build: add JSON info merge script
Script creates three different files types with different functions:

* map.json

Stored in each target/subtarget folder containing a mapping between
$(SUPPORTED_DEVICES) and JSON image info files. With this file a router
can automatically obtain a sysuprade firmware image.

    {
      "devices": {
        "netgear,r7800": {
          "info": "openwrt-ipq806x-generic-netgear_r7800.json",
          "sha256": "9d24baeba60c4f592838bf7fdf2a1f87b61d4cd1432953d2960315a37149b89a"
        },
        "r7800": {
          "info": "openwrt-ipq806x-generic-netgear_r7800.json",
          "sha256": "9d24baeba60c4f592838bf7fdf2a1f87b61d4cd1432953d2960315a37149b89a"
        }
      },
      "metadata_version": 1,
      "target": "ipq806x/generic"
    }

This mapping style allows device identifiers to change between releases, as it
happend between ar71xx and ath79.

Sha256sums are stored to verify file integrity, later more on that.

This files is stored in $(OUTPUT_DIR) and contains a list of all
targets where images exist for.

* targets.json

    {
      "metadata_version": 1,
      "targets": {
        "ar71xx/generic": {
          "path": "targets/ath79/generic",
          "sha256": "434bb3de0e5e0a1240a714d08360667daeb11f92b36cea4fb590046392f2f3a7"
        },
        "ath79/generic": {
          "path": "targets/ath79/generic",
          "sha256": "434bb3de0e5e0a1240a714d08360667daeb11f92b36cea4fb590046392f2f3a7"
        },
        "ipq806x/generic": {
          "path": "targets/ipq806x/generic",
          "sha256": "6afba149bc0b0f22e19ebb49e10d7e2a98e80286f84eafcd13b65384f71f401f"
        },
        "x86/64": {
          "path": "targets/x86/64",
          "sha256": "0f445ca9807d7f21624fff802f33320b30247eedfca636237317a144729c02e1"
        }
      }
    }

This mapping style allows device targets to change between releases, as it
happend between ar71xx and ath79. Devices searching for ar71xx are forwarded to
compatible deivces of ath79.

The workflow would be the following:

* Devices request a (tbd) signed versions.json file from the server
* Parse the file on device and optain the folder containing targets.json
* Request targets.json and optain path to target specific map.json
* Request map.json and optain device specific info file
* Parse info file images for a sysupgrade
* Request sysupgrade and flash device

The versions.json would be signed containing sha256sums of targets.json files.
This allows a chain of trust:

    versions.json -> targes.json -> map.json -> device_id.json -> sysupgrade.bin

The versions.json file could look like the following:

    {
        "versions": [
            {
                "name": "19.07-SNAPSHOT",
                "path": "releases/19.07-SNAPSHOTS",
                "sha256": "e659ccba1cd70124138726efabd9dd7a60ee220bf8471a91082c7e05488cac19"
            },
            {
                "name": "Snapshot",
                "path": "snapshots",
                "sha256": "7c84e6140f35c70d6fe71cd1ca8e5d98d9f88a887a4550adf137d4fd2a1f0ea6"
            }
        ],
        "metadata_version": 1
    }

* overview.json

Contains a mapping between device title(s) and device info files. Having
this file greatly simpifies firmware image retreiveal. A (web) client can
search trhough the mapping and show the results to an end user.

    {
      "devices": {
        " TL-WR840N v5": "targets/ramips/mt76x8/openwrt-ramips-mt76x8-tplink_tl-wr840n-v5.json",
        "7Links PX-4885 4M": "targets/ramips/rt305x/openwrt-ramips-rt305x-7links_px-4885-4m.json",
        "7Links PX-4885 8M": "targets/ramips/rt305x/openwrt-ramips-rt305x-7links_px-4885-8m.json",
        "8devices Carambola": "targets/ramips/rt305x/openwrt-ramips-rt305x-8devices_carambola.json",
        "8devices Carambola2": "targets/ath79/generic/openwrt-ath79-generic-8dev_carambola2.json",
        ...
      },
      "metadata_version": 1
    }

Signed-off-by: Paul Spooren <mail@aparcar.org>

@aparcar aparcar force-pushed the aparcar:CREATE_JSON branch from 47776f6 to 65f409d Sep 2, 2019

@aparcar

This comment has been minimized.

Copy link
Member Author

commented Sep 2, 2019

I created a proof of concept on how devices could find securely firmware upgrades.

"model": e("DEVICE_MODEL"),
"supported_devices": e("SUPPORTED_DEVICES").split(),
"target": "{}/{}".format(e("TARGET"), e("SUBTARGET")),
"title": list(

This comment has been minimized.

Copy link
@dangowrt

dangowrt Sep 12, 2019

Member

Wouldn't it be nicer to preserve the atomic info (VENDOR, MODEL, VARIANT) instead of only the already merged TITLE field? I imagine a structure like:

{
  "title":
    [
      { "vendor": "Arcadyan, "model": "ARV4520PW" },
      { "vendor": "Vodafone", "model": "Easybox 800" },
      { "vendor": "Airties", "model": "WAV-281" }
    ]
}

This comment has been minimized.

Copy link
@aparcar

aparcar Sep 14, 2019

Author Member

Thank you, I'll rework it.

@jow-

This comment has been minimized.

Copy link
Contributor

commented Sep 13, 2019

Where do these file end up? Are the JSON files really put next to the image files?
Does the merging really need to be part of buildroot? Can't this be done elsewhere?

@aparcar

This comment has been minimized.

Copy link
Member Author

commented Sep 14, 2019

It's not important to store the files next to the images, but putting them next to buildinfo and manifest files seemed reasonable to me. If you'd prefer so, I'd move the files to a different folder. However I'd really be in favor to have this files created by the buildbot. I see three use cases which all require up to date JSON files.

Removing the merging process from the buildroot be possible via the json overview but still, I'd much more like to see these files officially distributed at downloads.openwrt.org.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.