Merge branch 'master' into 17.01.4

README.md

@@ -5,18 +5,31 @@ Basic usage example for cooking a firmware for TpLink 4300:
 
 `./cooker -c ar71xx/generic --flavor=lime_default --profile=tl-wdr4300-v1`
 
+## Using cooker online with Chef
+
+cooker can be used also via [Chef](https://chef.libremesh.org/) web interface. Its source code can be found [here](https://github.com/libremesh/chef/). 
+
 ## Preparing the local environment
 
 ### Building in running system
 
-Before using lime-sdk, make sure your Linux system has the required dependencies installed. You might follow these instructions (look for _Examples of Package Installations_) https://lede-project.org/docs/guide-developer/install-buildsystem
+Before using lime-sdk, make sure your Linux system has the required dependencies installed. 
+
+Install build dependencies, for example on a Debian/Ubuntu based Linux distribution install the following packages:
+
+```
+sudo apt-get install subversion zlib1g-dev gawk flex unzip bzip2 gettext build-essential libncurses5-dev libncursesw5-dev libssl-dev binutils cpp psmisc docbook-to-man wget git
+```
+
+For other systems, you might follow these instructions (look for _Examples of Package Installations_) https://lede-project.org/docs/guide-developer/install-buildsystem
 
 ### Building in docker container
 
 Install [Docker](https://www.docker.com/get-docker) and run the following command:
 
-	docker build -t cooker .
-	docker run -it cooker -v "$(pwd":/app --<parameters>
+	cd lime-sdk
+	sudo docker build -t cooker .
+	sudo docker run -v "$(pwd)":/app cooker --<parameters>
 
 ## Targets, profiles and flavors
 LibreMesh can be used on many different devices (target and profile) and can be packed in many different ways (flavors), depending on your needs. To this end, it is important to choose the right options for building your firmware.
@@ -69,13 +82,25 @@ The standard steps to generate a firmware would be: firstly build and secondly c
 
 However, cooker is smart enough to detect the missing steps and transparently execute them. If we choose to cook before building, it will automatically build before cooking. Therefore, for debugging purposes it is better to execute the steps separately.
 
+
 On the other hand, if you do not want to build locally (since it requires some special software installed on your Linux machine), you can just cook using the online precompiled binaries.
 
 ##### Building locally or fetch remote?
 Cooker can locally build the LibreMesh packages or fetch the remote precompiled ones. For most users there is no real need for building, since using the remote ones might be a better (and fast) option. To remotelly fetch the packages the special option _--remote_ must be used when cooking, like this:
 
 `./cooker -c ar71xx/generic --profile=tl-wdr4300-v1 --flavor=lime_default --remote`
 
+#### Using custom SDK and/or IB files
+Custom local SDK and IB files can be used (instead of fetching official LEDE sources). Must be specified before building or cooking ("-b" or "-c").
+
+`./cooker -f`
+`./cooker -i ar71xx/generic --ib-file=myOwnImageBuilder.tar.xz --sdk-file=myOwnSDK.tar.xz`
+`./cooker -b ar71xx/generic --force-local`
+`./cooker -c ar71xx/generic --profile=tl-wdr4300-v1 --flavor=lime_default --force-local`
+
+Do not forget to use _force-local_ option to use your own SDK target packages (kernel signature will be different from remote sources).
+
+
 ## Communities
 It is not required to create and/or use a community profile. But if you are constantly cooking LibreMesh for your network and you need a specific configuration, you might want to spend some time creating a community profile. This will make things easier and better coordinated for your community.
 
@@ -237,3 +262,87 @@ Time to time, if you want to update the code with the official one you might add
 
         J=1 V=s ./cooker -b ar71xx/generic --profile=tl-wdr3500-v1
 
+## Testing on QEMU
+
+While developing new features, or just testing out fixes, being able to see them in action without having to reflash a device can be useful. To
+achieve this you can spin a [QEMU](https://en.wikipedia.org/wiki/QEMU) virtual machine and boot the image with your edits.
+These instruction are based on [LEDE documentation](https://lede-project.org/docs/guide-developer/test-virtual-image-using-armvirt) but are a bit more specific to LibreMesh building process.
+
+First of all you need to create your cooked version of LibreMesh firmware for the `armvirt` target, see [up here](#preparing-the-local-environment).
+
+    cd lime-sdk
+    ./cooker -c armvirt/generic --flavor=lime_default --update-feeds
+
+Once `cooker` finishes to build the image you'll find the needed files in the `output` folder of `lime-sdk`, they will be located in a subfolder
+accordingly to the architecture and profile chosen. The interesting files are:
+
+ * lede-17.01.2-lime-XXXX-zImage
+ * lede-17.01.2-lime-XXXX-root.ext4.gz
+
+Uncompress `lede-17.01.2-lime-XXXX-root.ext4.gz` using `gunzip -k lede-17.01.2-lime-XXXX-root.ext4.gz`
+
+Now you need to install qemu in order to boot the image, usually it's available inside the repositories of the distribution. Here some quick links
+documenting how to install it on [Debian](https://wiki.debian.org/QEMU) or [ArchLinux](https://wiki.archlinux.org/index.php/QEMU).
+
+Note that if you
+want to use an image built for arm you should have `qemu-system-arm` command available, often provided by `qemu-system-arm` or `qemu-arch-extra` package.
+
+Now it's time to spin the virtual machine.
+
+
+### Using plain QEMU
+Plain qemu can be launched straight from the command line, if you don't need to access LibreMesh web interface and just want to have a shell you can issue
+
+`qemu-system-arm -nographic -M virt -m 64 -kernel lede-17.01.2-lime-XXXX-armvirt-zImage -drive file=lede-17.01.2-lime-XXXX-armvirt-root.ext4,format=raw,if=virtio -append 'root=/dev/vda rootwait'`
+
+Press enter and you will find yourself inside the VM booted.
+
+You can also have access to the web interface configuring a tap device on the host as follows.
+
+TODO
+
+### Using Virt-Manager
+
+[VirtManager](https://github.com/virt-manager/virt-manager) is an higher level way to deal with virtualization using libvirt. Libvirt supports several
+virtualization technologies, not only Qemu. It's a quick'n'easy way to setup a test environment.
+
+Many distributions provide packages for Virt-Manager, you have to install it together with qemu:
+
+* **ArchLinux**: `sudo pacman -S virt-manager`
+* **Debian** and **Ubuntu**: `sudo apt-get install virt-manager`
+
+If you plan to use networking functions, like accessing the web interface, you'll need to install also `iptables` and `ebtables` packages.
+
+The setup is a bit longer but is persistent and you will only need to rebuild the images from `cooker` and re-spin the VM to see your changes.
+
+What you need to do is to start the libvirtd and virtlogd daemons (if not already started start them with `sudo systemctl start libvirtd.service virtlogd.service`), open Virt-Manager and ensure you are connected to the
+libvirt socket.
+
+Now you have to create a new virtual machine: click on *File/New Virtual Machine*, select *Import existing disk image* and choose the *arm* architecture under *Architecture options* and *virt* machine type.
+
+Taking arm as an example you'll have to choose (clicking *Browse* and *Browse Local* buttons) the `lede-17.01.2-lime-XXXX-armvirt-root.ext4` file as storage disk and the `lede-17.01.2-lime-XXXX-armvirt-zImage` file as Kernel path. Insert `root=/dev/vda rootwait` as Kernel args. You can leave *OS type* as *Generic*.
+
+Assign resources (64 MB of RAM memory should be enough) and, under *Network selection*, choose `NAT` as network type. In this way you will be able to connect to the web interface and the device will have internet access.
+
+If you started the VM at this point, it will hang, you can shut it down from the menu *Virtual Machine/Shut Down/Force off*.
+
+What is missing is to change the disk bus mode: open the VM windows without starting the VM, click on *View/Details* open the *SATA disk 1* tab and change *Disk bus* from *SATA* to *VirtIO*. You should now be able to start the VM.
+
+Libvirt automatically create a bridge interface for you to which the VMs are connected to. Assign your bridge device a network address inside LibreMesh subnet
+and you should be good to go, something like:
+
+`sudo ip address add 10.13.246.1/16 dev virbr0`
+
+Or get one via DHCP with: `sudo dhcpcd --metric 9999 virbr0` or `sudo dhclient -e IF_METRIC=9999 -i virbr0`
+
+Should be good, you can be sure about the address opening the VM and issuing an `ip address show dev br-lan` once the interfaces are correctly set-up.
+
+You can access the router web interface in a browser with the router IP or anygw IP which could be something like `10.13.0.1`
+
+If the router has just one ethernet interface, like in our VM, LibreMesh by default doesn't use that interface as WAN. If you need the router to have access to the internet, using the VM console interface edit `/etc/config/lime` adding a specific interface configuration like:
+
+    config net manualwan
+        option linux_name 'eth0'
+        list protocols 'wan'
+
+applying the new settings with the command `lime-config; service network reload`. A drawback is that in this way the web interface can not be accessed anymore because of LibreMesh firewall blocking connections incoming from the WAN interface.

cooker

@@ -141,7 +141,7 @@ cook() {
     echo "-> Cooking firmware image"
     join_packages ${!flavor}
     echo "--> Selected extra packages:$PKG"
-    make -C $ib image $profile PACKAGES="$PKG" EXTRA_IMAGE_NAME="$fw_extra_name" BIN_DIR="$output_dir" $ib_files $force_no_update $files_remove
+    make -C $ib image $profile PACKAGES="$PKG" EXTRA_IMAGE_NAME="$fw_extra_name" BIN_DIR="$output_dir" FILES="$ib_files" $force_no_update $files_remove
     ERROR=$?
     if (($ERROR != 0)); then
       echo -e "\n-> Firmware not cooked, something wrong happened on the ImageBuilder compilation process"
@@ -355,9 +355,11 @@ download_sdk() {
 unpack_sdk() {
     local file="$1"
     local output="$2"
+
     [ -d $output/sdk ] && [ -z "$force_install" ] && {
         echo "-> SDK for $target already exist, doing nothing. Use --force to reinstall"
     } || {
+        [ ! -d "$output" ] && mkdir -p "$output"
         echo "-> Unpacking SDK to $output"
         tar xf $file -C $output/ && {
             [ -d $output/sdk ] && rm -rf $output/sdk
@@ -396,6 +398,7 @@ unpack_ib() {
      [ -d $output/ib ] && [ -z "$force_install" ] && {
         echo "-> ImageBuilder for $target already exist, doing nothing. Use --force to reinstall"
      } || {
+        [ ! -d "$output" ] && mkdir -p "$output"
         echo "-> Unpacking ImageBuilder to $output"
         tar xf $file -C $output/ && {
            [ -d $output/ib ] && rm -rf $output/ib
@@ -480,7 +483,7 @@ while true; do
         ;;
         --update-feeds)
             update_feeds
-            break
+            shift
         ;;
         --no-update)
             no_update=1
@@ -594,6 +597,10 @@ flavor=${flavor:-$default_flavor}
 profile=${profile:-Default}
 
 [ "$local_install" == "1" ] && {
+  [ -z "$target" ] && {
+    echo "-> Target must be specified for local install (i.e -i ar71xx/generic)"
+    exit 1
+  }
   [ -n "$sdk_path" ] && unpack_sdk "$sdk_path" $release/$target
   [ -n "$ib_path" ] && unpack_ib "$ib_path" $release/$target
   [ -z "$sdk_path$ib_path" ] && {

feeds.conf.default

@@ -1,7 +1,7 @@
 src-git base https://github.com/lede-project/source.git;v17.01.4
 src-git packages https://git.lede-project.org/feed/packages.git;lede-17.01
 src-git luci https://git.lede-project.org/project/luci.git;lede-17.01
-src-git routing https://git.lede-project.org/feed/routing.git;master
+src-git routing https://git.lede-project.org/feed/routing.git;lede-17.01
 src-git telephony https://git.lede-project.org/feed/telephony.git;lede-17.01
 
 src-git libremesh https://github.com/libremesh/lime-packages.git;17.06

libremesh.repositories.conf

@@ -2,3 +2,4 @@ src/gz libremesh http://repo.libremesh.org/current/packages/{ARCH}/libremesh
 src/gz libremap http://repo.libremesh.org/current/packages/{ARCH}/libremap
 src/gz limeui http://repo.libremesh.org/current/packages/{ARCH}/limeui
 src/gz lm_routing http://repo.libremesh.org/current/packages/{ARCH}/routing
+src/gz lm_profiles http://repo.libremesh.org/network-profiles/