Computing prefs 2.0

[davidpanderson]

The current prefs system can't handle some plausible requirements,
e.g. disabling a particular GPU when the computer is in use.
This is a proposal for a more general prefs system, which I'll call P2.

P2 is intended to be as general as possible,
i.e. it allows any conceivable preferences to be expressed.
That way we only have to do this once.

Like the current system, P2 uses XML to represent preference sets.
It would be possible to use another data representation language such as JSON,
or to use an embedded programming language like Tcl,
but I think XML is the best choice.

A P2 spec consists of a sequence of "clauses".
Each clause consists of a "condition" specifying when it's in effect,
and one or more preferences.
For example, the preference mentioned above would be represented as:

<clause>
    <condition>
        <not><idle_time_more_than>120</idle_time_more_than></not>
    </condition>
    <prefs>
        <gpu>
            <type>nvidia</type>
            <number>1</number>
            <enable>0</enable>
        <gpu>
    </prefs>
</clause>

If no condition is given, the clause is always in effect.
Typically a spec will begin with such a clause.

The clauses are evaluated in order.
Later clauses (if their conditions are met) override earlier ones.
For example, if we want to enable GPU use from 5PM to 7AM,
regardless of idleness, we'd add another clause

<clause>
    <condition>
        <daytime>17-7:<daytime>
    </condition>
    <prefs>
        <gpu>
            <enable>1</enable>
        </gpu>
    </prefs>
</clause>

---

Conditions

P2 conditions are boolean expressions.
The primitives are

<idle_time_more_than>
true if mouse/keyboard input in last N seconds
<on_batteries>
true if running on batteries
<program_running>name</program_running>
true if "name" is executing
<daytime>X:Y</daytime>
true if day is in range X and time is in range Y
<cpu_usage_more_than>X</cpu_usage_more_than>
non-BOINC CPU usage exceeds X
<external>X</external>
the GUI has told us that condition "X" holds.
<battery_charge_more_than>
<battery_temperature_more_than>
<on_wifi>
connected via WiFi (mobile)

Boolean expression can be formed using
<not>X</not>
negation of X
<or>...</or>
or of 2 or more expressions
<and>...</and>
and of 2 or more expressions

Expressions can be nested.

---

Prefs

<max_ncpus_pct>
<cpu_usage_limit>
<gpu>
    enable or disable one or more GPUs
<ram_max_used_pct>
<vm_max_used_pct>
<network>
    enable or disable all network communication
<file_xfer>
    enable or disable file transfers
<max_vm_tasks>
    limit the # of running virtual machine tasks

The following are typically static but it doesn't hurt to make them dynamic.

<leave_apps_in_memory>
<work_buf_min_days>
<work_buf_additional_days>
<cpu_scheduling_period>
<disk_interval>
<disk_max_used_gb>
<disk_min_free_gb>
<disk_max_used_pct>
<daily_xfer_limit_mb>
<daily_xfer_period_days>
<dont_verify_images>
<max_bytes_sec_down>
<max_bytes_sec_up>

---

Implementation, Phase 1

The goal of this phase is to make the features of P2
available to users who are willing to write XML specs.
Steps:

a) define a C++ data structure to represent a P2 preference set.

b) write code that parses an old-style specs (global_prefs.xml, cc_config.xml)
into this data structure.

c) enforce prefs based on the new data structure, rather than GLOBAL_PREFS.
Need to evaluate prefs when

• time of day (if any conditions use it)
• app starts/stops running
• idle time exceeds a limit or returns to zero
• etc.

This is entirely within the client; I should do this part.

---

Implementation, Phase 2

The goal of this phase is to provide GUIs for editing P2 prefs.

In both web and desktop, the primary GUI should be choosing a preset
(standard, max computing, green, etc.).
An "Advanced" button leads to a full-featured prefs editor, which lets you

• create/delete/reorder clauses
• edit conditions (including boolean expressions)
• edit prefs

This editor should be implemented using HTML and Javascript.
There are various JS frameworks that could be used.

In the desktop case (i.e. BOINC Manager or Android GUI)
the Advanced button should take you to a prefs-editor web page,
served from the BOINC web site.
The "Save" button writes the resulting XML to a local file.
In other words, we should NOT implement P2 prefs editors
in the WxWidgets and Android GUIs.

Developing the editor is a self-contained task and anyone
with HTML/JS experience could do it.
Integrating it with the BOINC web code and with the GUIs
would be a small amount of additional work.

[AenBleidd]

In other words, we should NOT implement P2 prefs editors
in the WxWidgets and Android GUIs.

Why not?
Personally I can do this task for Desktop Manager part.

[Ferroin]

In other words, we should NOT implement P2 prefs editors
in the WxWidgets and Android GUIs.

Why not?
Personally I can do this task for Desktop Manager part.

If it's done as a pure HTML/CSS/JS design, it can then be trivially used on almost any platform without needing separate code-bases, including possibly being directly usable for the web-based preferences editor, and in turn means that any new preferences need to make a change in exactly one location in the code to have them usable by everybody.

Whether or not that is worth it is another question of course.

[AenBleidd]

Whether or not that is worth it is another question of course.

Personally I (and many other users) will be confused: why some web-page is shown while I want to change local settings? As far as I know, no other application do like this

[ChristianBeer]

I took the liberty to add some markup to the initial proposal because it was hard to read with almost all of the HTML tags missing.

Just some quick thoughts:

• when using XML and ordering the clauses it is difficult to use an external XML library because they tend to do some sorting themselves. It might be better to add an integer based ordering system to the clauses
• as we found out in previous discussions the naming is important; I would rather call the clauses rules that have conditions that are applied to resources. That way it might be easier for the end-user to understand
• there must already be some XML based dialects in the wild that describe such rules and might be suitable for our case. Usually they already have some kind of visual style editor (IFTT tools come to mind).
• It might be beneficial to look at this from the other end (what the end-user sees) and design a visual editor first, then develop a language that fits. If we do it the other way whoever builds the visual part has to work around problems in the XML dialect and the visual editor might feel clunky because of this.

[davidpanderson]

I started implementing this in the dpa_prefs2 branch;
see also https://boinc.berkeley.edu/trac/wiki/Prefs2

[Toby-Broom]

Can you add a ram limit the prefs, so I can simulate less ram that the detected ram like with CPU's. My use case is running multiple instances of boinc but they both think they have the full ram ammount

[truboxl]

Can you add a ram limit the prefs, so I can simulate less ram that the detected ram like with CPU's. My use case is running multiple instances of boinc but they both think they have the full ram ammount

1. These settings can't fulfill your needs?

When computer is in use, use at most X %
When computer is not in use, use at most Y %
Page/swap file: use at most Z %

2. On another note, can the battery percentage rules from Android merged with the Desktop client?

[Toby-Broom]

Yeh, I guess I was thinking more about a real number like the config method not the client method.

Everything gets complex since the project I use does not set the rsc_memory_bound properly

[JStateson]

The example at the top of this thread shows the procedure

<clause>
    <condition>
        <not><idle_time_more_than>120</idle_time_more_than></not>
    </condition>
    <prefs>
        <gpu>
            <type>nvidia</type>
            <number>1</number>
            <enable>0</enable>
        <gpu>
    </prefs>
</clause>

This would be quite useful for Boinc gridcoin miners where a GPU has failed and needs to be immediately excluded from the pool of available GPUs. Along with, or even preceding this preference change, could be the useful tool: boinccmd --coproc <id> <enable | disable> I am currently trying to implementing this (without a client restart) in my "special boinc app"
This would allow 3rd party apps such as BoincTasks to use its existing "rules" to enable or disable coprocessors. I started a "wish list" for this very thing here https://forum.efmer.com/index.php?topic=1394.msg8047#msg8047

[Aurum420]

BOINC grids such as WCG & LHC are mixing very different kinds of projects under the same Preferences. E.g., WCG now has two projects that require a fast turn around; the serial FAH2 and long ARF with sparse checkpoints. Clients need more effective ways to control their work.
Please add BOINC commands & that tell the server what to do.

<app_config>
<app>
    <name>ATLAS</name>
    <!-- Xeon E5-2699 v4  22c44t  32 GB RAM L3 Cache = 55 MB  -->
    <maintain>18</maintain>
    <max_concurrent>16</max_concurrent>
</app>
<app_version>
    <app_name>ATLAS</app_name>
    <plan_class>native_mt</plan_class>
    <avg_ncpus>1</avg_ncpus>
    <cmdline>--nthreads 1</cmdline>
</app_version>
<app>
    <name>sixtrack</name>
    <maintain>9</maintain>
    <max_concurrent>6</max_concurrent>
</app>
<app>
    <name>Theory</name>
    <maintain>44</maintain>
</app>
<app>
    <name>CMS</name>
    <maintain>0</maintain>
</app>
</app_config>

And even better would be.

<app_config>
<app>
    <name>ATLAS</name>
    <!-- Xeon E5-2699 v4  22c44t  32 GB RAM L3 Cache = 55 MB  -->
    <priority>1</priority>
    <max_concurrent>16</max_concurrent>
</app>
<app_version>
    <app_name>ATLAS</app_name>
    <plan_class>native_mt</plan_class>
    <avg_ncpus>1</avg_ncpus>
    <cmdline>--nthreads 1</cmdline>
</app_version>
<app>
    <name>sixtrack</name>
    <priority>3</priority>
</app>
<app>
    <name>Theory</name>
    <priority>2</priority>
</app>
<app>
    <name>CMS</name>
    <priority>0</priority>
</app>
</app_config>

<maintain> is useful for very large WUs like LHC's ATLAS that are ~250 MB. Since internet providers choke the upload speed it takes a long time for an ATLAS completed WU queue to clear before server knows to send more work. By maintaining a couple extras in the wings work continues uninterrupted. Zero means don't ever send me this class of WU even if "If no work for selected applications is available, accept work from other applications? yes" is selected. E.g., LHC CMS has no native Linux ap and I do not run Oracle's VirtualCatBox.
<priority> allows one to keep serial projects like FAH2 always running and to avoid time-slicing.

[DennisvdPluijm]

Not sure if I understand this feature request correctly, so let me check with a real-world example case (and the reason I found this request):
I have a laptop and would like to configure BOINC such that it will use 2 (of 8) cores when plugged in, that it will use 4 cores if 1) plugged in and 2) not in use, and that it will use only 1 core if running on battery.
As far as I know and have searched, it is not possible to configure BOINC this way right now. Would it be possible with this feature?
If so, I greatly support it!

[AenBleidd]

@DennisvdPluijm, yes

[talregev]

Please add device name to windows / linux / mac clients.

[brevilo]

Just a few comments from the trenches (based on decades of first-hand gripes):

• since you stick to XML, please define a proper XML schema (XSD) that can be used to validate any incoming/processed XML.
• this schema should define the canonical truth, so if you stick to using a custom XML parser, please make sure it adheres to the schema fully. All other non-codified behavior should be considered a bug in the parser.
• ideally, when coming across "invalid" XML, try and self-heal it centrally as much as possible, such that it doesn't propagate from project to project (a known issue).
• please version the schema and support schema version transitions with suitable tooling for automatic upgrades (i.e. when projects upgrade their BOINC server version).

Thanks