Emailed comment from the Department of Energy

[h-m-f-t]

Comment No.	Location	Comment (and type*) or Recommended Edit	Rationale
1	Pg. 1, §1	Critical: A Binding Operational Directive (BOD) is an ill-suited vehicle to mandate this type of policy. DOE does support the development of vulnerability disclosure policies, however, DOE believes this should be done through an alternative policy deliberation process that allows for more discussion and thoughtful review of costs, operational and security impacts, and exceptions.	Per FISMA, the purpose of a BOD is "to safeguard Federal information and information systems from a known or reasonably suspected information security threat, vulnerability, or risk" yet the BOD does not directly identify a "known or reasonably suspected information security threat, vulnerability, or risk". DOE has seen no data to suggest a lack of VDP has significantly hindered federal cybersecurity, particularly with a significantly aggressive timeline (e.g. 15 days). Given this, the development of VDP through the policy process would be preferred.
2	Pg. 1, §1	Substantive: Request the risk rationale and cost-benefit analysis supporting the promulgation of this draft proposed policy.	Appropriate use of the BOD process – Due to the very high cost and mission impact of imposing a Binding Operational Directive, this regulatory vehicle should be held to a very high bar for appropriateness and prioritized risk reduction. The DHS does not disclose the process used to determine that this proposed directive, as the most critical risk, has no risk reduction target, and the metrics associated with this directive are aligned to insure reporting, not the reduction of risk.
3	Pg. 1, §1	Critical: Insert exclusion clause for out of scope systems such as classified National Security Systems (NSS).	The DOE has a unique set of missions that require coordination with both the DoD and the Intelligence Community, that requires additional consideration. NIST provides the following definition for NSS: "any information system (including any telecommunications system) used or operated by an agency or by a contractor of an agency, or other organization on behalf of an agency—(i) the function, operation, or use of which involves intelligence activities; involves cryptologic activities related to national security; involves command and control of military forces; involves equipment that is an integral part of a weapon or weapons system; or is critical to the direct fulfillment of military or intelligence missions (excluding a system that is to be used for routine administrative and business applications, for example, payroll, finance, logistics, and personnel management applications); or (ii) is protected at all times by procedures established for information that have been specifically authorized under criteria established by an Executive Order or an Act of Congress to be kept classified in the interest of national defense or foreign policy." Based on this definition, DOE will exclude systems and services related to nuclear technology, other technologies deemed sensitive or related to national security, and all systems and services that interact directly with DoD or the Intelligence Community from the scope of this BOD and will make note of such exception within its policy and handling procedures.
4	Pg. 3, §1	Substantive: Remove all “Bug Bounty” references and discussion.	Discussion of “Bug Bounty” is not relevant to the direction provided in the draft proposed policy. Rather the current language only provides notification of intent of OMB to convene an unnamed group of agencies to consider “leveraging bug bounty programs.”  Inclusion of Bug Bounty references is confusing and generally reinforces the misconception that VDP and Bug Bounty are the same.
5	Pg. 4, §1	Substantive: Extend .gov registrar information from 15 business days to 60 days.	While this may be accomplished for the “DOE.GOV” domain, the vast majority of the Department's “.GOV” footprint is managed in the Management & Operating (M&O) contract space to include .org and .edu addresses. For example, the DOE has a minimum of 35 second-level .gov domains and 3,878 subdomains. The inferred “capability to receive unsolicited reports about potential security vulnerabilities” for the diverse, federated DOE community will require additional time considering the Department's diverse internet domain footprint.
6	Pg. 4, §3	Critical: Recommend rewording the requirement to direct Agencies to develop a plan for a VDP and handling procedures within 180 days of BOD issuance and begin implementing their VDP for internet facing systems and services according to their plans' prescribed timeline. As written, this BOD will have significant cost implications considering the DOE's federated nature and broad internet presence. Given a lack of clear correlation between unreported vulnerabilities and federal risk, it is hard to justify the resources necessary to meet the timelines and requirements as written. Should agencies have the opportunity to develop their respective plans for VDP within 180 days and have additional time to implement, this would allow each Agency to take their respective challenges into account without significant unbudgeted costs.	This is an unfunded mandate with sweeping impact to Agency contracts, which will be made significantly more difficult given the tight timeline. If Agencies were instead directed to develop a plan for implementing VPD within 180 days (as opposed to having their VDP in place within 180 days), they could assess and consider total costs and impacts as implemented. Notably, the DOD model required 11 full-time equivalents (FTEs) to implement a similar VDP - replicating this at DOE is not included in the existing budget allocations. Due to the nature of the DOE's organization, National Laboratories are often contractor owned, contractor operated (COCO) or government owned, contractor operated (GOCO), each organization type requiring strict contracts and agreements to manage operations and costs. Applicability of the BOD to federated elements – an open question remains as to the applicability of the BOD process to government owned, contractor operated M&O institutions. Contract modifications for either organizational model will be required for each National Laboratory to implement the requirements as prescribed. Extensive contract modifications will be lengthy and necessary with major cost implications, which are not currently budgeted for. In addition the cost in additional FTEs to execute VDP requirements will likely be higher on Laboratories as the rates will be based on contractor and vendor wage rates.
7	Pg. 4, §3	Critical: If the request to require a plan for VDP and handling procedure development within 180 days is rejected, DOE requests extending the requirement for publication of a VDP from 180 calendar days to one year of directive issuance.	Accelerated DOE policy publication process takes an average of 7-8 months making it difficult to comply with the current timeline as provided. Due to the DOE's federated environment, proposed policies must undergo extensive program office reviews to ensure alignment with contractual and legal requirements.                                                                                                          In addition, ensuring clear guidance on how to submit vulnerability reports requires that tools and resources to facilitate that process be identified, procured, and ready for use by the time the policy is published. Ensuring the proper tools and resources are in place will require more than 180 calendar days to organize and confirm, and as mentioned, extensive contract modifications will be needed for existing laboratories and operations, which can take over a year to finalize and go into effect. The compressed timeline for policy issuance, underlying timelines for implementation, and expanded scope may force DOE to outsource vulnerability disclosure reporting, handing procedures, and the program itself to third-party bug bounty platforms at greater cost in order to meet the compliance schedule instead of building an internal program the DOE can manage and mature. The cost and process impact to DOE sites will have a material impact on DOE field elements. This impact will be non-proportionally acute for the smaller sites.
8	Pgs. 4, 5, 6; §3.a.i., 5, and 8.a.iv	Critical: Request clarification on definition of internet-accessible production system or service to handle unique contractual relationships and obligations.	There is ambiguity over what internet accessible systems and services are in scope and whether that includes internet-accessible systems and services owned and controlled by an Agency or any internet-accessible system or service used by an Agency or which contains Agency information. Similar to other Agencies, the DOE may use internet-accessible systems and services (e.g., cloud service providers and software as a service) to execute operations but those systems and services may be leased from, contractually owned, and provided by a third party service provider or partnering institution. The DOE's VDP cannot sanction testing of third party-owned assets or offer any legal protections. This is of particular concern to the DOE given its unique structure whereby Laboratories often share systems and services with external partners and third parties, or where information on such systems is not owned by the DOE but is research owned by other institutions that is being held on DOE assets.
9	Pgs. 4, 5, §3.a.iii, 3.a.v	Critical: If the request to require a plan for VDP and handling procedure development within 180 days is rejected, DOE requests extending the requirement for publication of vulnerability disclosure handling procedures from 180 calendar days to one year of directive issuance.	Accelerated DOE policy publication process takes an average of 7-8 months making it difficult to comply with the current timeline as provided. Due to the DOE's federated environment, proposed policies must undergo extensive program office reviews to ensure alignment with contractual and legal requirements.                                                                                                                                                                                                                                                                                          Requirements to describe how vulnerability reports may be submitted, including where and what information to include, requires a clear submission process and access to the proper tools or resources needed to execute that process. Determination and deployment of such processes or supporting tools and resources are likely to require more than 180 days to complete, which makes documenting them in formal policy difficult to achieve.
10	Pg. 5, §3.a.v	Substantive: Add a clause that enables agencies to define the level of remediation detail provided based on the sensitivity in VDP rules of engagement and as applicable to mission delivery.	Agencies should have the flexibility to determine the level of information that should be shared with reporters depending on the sensitivity of the vulnerability. With a range of missions from open science to nuclear, the level of transparency should be appropriate for the program risk.
11	Pg. 5, §3.b.ii	Critical: Agencies should not be restricted in their ability to register authorized parties for testing with minimal burden of personal information and limit the participation of others. Due to the potential sensitivity of IT assets, information, and operations, the ability to flag potential testers in alignment with each Agency's mission should be provisioned.	Although a noble goal to allow anyone, anywhere to test federal systems, the additional resources needed to respond to all un-vetted testers will have a significant impact on cyber operations. Without having a method to vet and communicate with reporters, agencies have no way to constrain resource loading (e.g. on cloud usage) with direct impact to staff (e.g., SOC resources on false positives) and funding levels. It will also make it harder to separate malicious actors from authentic security researchers. For example allowing registered users to use a valid VPN for testing, tracking, and flow management.
12	Pg. 5, §3.b.iii	Critical: Add clause to enable agencies to define a required time limited response period as part of their "good faith" criteria that may not be extend past 180 days.	By only allowing for a 'request for a reasonably time-limited response period', agencies have even less control on how to prioritize and address reported vulnerabilities and may expend limited resources to remediate all vulnerabilities immediately to meet the time-limited response period prior to disclosure to other parties. DOE is a unique agency that has significant interactions with DoD and the Intelligence Community. Given that the current BOD also does not allow for limiting testers to vetted reporters, there is significant risk that this would increase DOE's exposure to malicious actors if reporters are given the ability to immediately report vulnerabilities to third parties as well. For example, since reporters are not precluded from submitting their identified vulnerability to 'others' and can do so at any time, even with a request by an Agency for a time-limited response time, this could encourage active testing to then provide criminal or malicious actors with information on methods for exploitation. Since 'others' is not defined, any third party could be contacted by the reporter, including nation-state actors and registered or known federal hackers and criminal actors. With the requirement under 3.b.v to be transparent about remediation efforts, that same information could be presented to 'others' who could then execute an exploit on the vulnerability either prior to remediation or immediately after - having found a new method of exploiting the vulnerability. It is premature for this directive to identify a time restricted period because agencies do not know the volume of reports they will receive and the amount of time it will take to respond and remediate. Without this information agencies cannot realistically determine what a reasonable defined time would be and it will vary from agency to agency. However, while the time limited response period will vary according to each agency the volume of vulnerabilities received and the severity of those vulnerabilities, because agencies will not be able to exceed 180 days the spirit and goal of VDP will remain intact.
13	Pg. 6, § 7	Editorial: Recommend limiting initial scope to include only domains and systems/services owned by an Agency.	Some DOE sites use .edu and other domains, based on their contract and research relationships, which complicates the potential process. Scope of this draft proposed policy is explicitly limited to “.gov” domains. This will not capture DOE M&O footprint and will incentivize movement away from the “.gov” domain. Additionally, there are a number systems and services used by DOE contractors but not owned by any DOE elements or program offices. Should all systems and services used by an Agency be in scope, this will have significant legal ramifications should unrelated third parties be exposed to testing through an Agency VDP.
14	Pg. 7, §9.a-c	Substantive: Recommend identifying preferred or required method for submission of reports to CISA.	Method for how an agency should report any valid or credible reports, or escalate reports to CISA for action or visibility is not identified. To ensure consistent reporting and compliance, and that all reporting is channeled to the proper point of contact for escalation and consideration, a specific mailbox or reporting method with required. The method should also identify core data fields needed by CISA. Procedures setting expectations for CISA response and review of submissions is also requested to ensure timely and transparent communication with agency stakeholders, system or service owners, and reporters, per requirement 3.a.v.
15	Pg. 7, §9.c	Substantive: Strike section entirely.	Current language is too broad and vague. "Any situation that is deemed helpful or necessary" is too open ended and almost any instance could be construed to fit under that definition. With an inundation of reports that can broadly align themselves to this criteria, agencies will expend significant resources to review and address all reported potential vulnerabilities (or even customer complaints).
16	Pg. 7, §10	Substantive: Execution of requirements under item #10 will require an update to the FISMA CIO Metrics through OMB. Coordination with OMB required but status unknown.	FISMA quarterly reporting is based on requirements set forth by OMB through the FISMA CIO Metrics. Inclusion of additional reporting requirements in FISMA quarterly submissions will require alignment with OMB and the FISMA CIO Metrics and recommend removing an equal number of other metrics to keep reporting burden neutral.
17	Pg. 7, §10	Critical: Recommend extending requirement for updated FISMA quarterly reporting metrics from 270 calendar days from directive issuance to 270 days from issuance of Agency policy and handling procedures.	Compliance with associated requirements will take time to implement. Following publication of an Agency policy and handling procedures, the DOE's federated departmental elements, program offices, sites, and national laboratories will need time to not only incorporate the policy and procedures but also execute them properly. Gathering metrics with integrity for inclusion in FISMA reporting will thus require more time than the prescribed timeline.

* Critical: Fundamental error or omission, Substantive: Preferred change, Editorial: Grammatical or stylistic

[afeld]

agencies have no way to constrain resource loading (e.g. on cloud usage)

TTS mitigates this problem in our Bug Bounty by saying in our policy that "Network denial of service (DoS or DDoS) tests [are not authorized.]" In the years we have been running the program, we have no reason to believe that security researchers have contributed significantly to network load on our systems.

It will also make it harder to separate malicious actors from authentic security researchers.

Unless you are providing them with special access, that vetting/determination is not necessary. A vulnerability report is a vulnerability report. The point of a vulnerability disclosure program is to provide a mechanism for good-intentioned actors to make these reports, not to catch malicious actors.

[zmanion]

[Responding to DOE comments 1 and 2]

...the BOD does not directly identify a "known or reasonably suspected information security threat, vulnerability, or risk".

Perhaps the BOD language could be more clear, but the proposition is straight forward: It is reasonable to suspect that agencies have internet-facing services with vulnerabilities that are unknown to the agency. Risk is a bigger question, primarily due to the wide variety of possible impacts to agency data and missions. Part of the threat factor is also straight forward: Adversaries scan internet-facing services and identify vulnerabilities.

DOE has seen no data to suggest a lack of VDP has significantly hindered federal cybersecurity

Functional measurement of cybersecurity is admittedly difficult. Falling back to the proposition: Experience from many bug bounty programs and VDPs, including the DOD VDP [1], shows that despite other efforts, internet-facing services are deployed and operated with vulnerabilities. If there is lack of evidence showing that not having VDP has not hindered security, there is data showing that having VDP has improved security, by removing internet-facing vulnerabilities.

[1] https://hackerone.com/deptofdefense

[zmanion]

[Responding to DOE comment 3, also partly 8]
The BOD directs agencies to determine appropriate scope for their VDPs, so there should be no problem with DOE and other agencies excluding particularly sensitive services. On the other hand, do you really not want to know about a vulnerability in a sensitive system exposed to the internet? Or a classified system or data accidentally exposed to the internet?

[zmanion]

[Comment 12]

Given that the current BOD also does not allow for limiting testers to vetted reporters, there is significant risk that this would increase DOE's exposure to malicious actors if reporters are given the ability to immediately report vulnerabilities to third parties as well.

Malicious actors are generally not constrained by VDP policy or other law or regulation, so a VDP doesn't affect (increase) exposure to malicious actors. If a VDP removes internet-facing vulnerabilities, that would reduce exposure to malicious actors. Malicious actors can identify, exploit, publish, sell, or tell other parties about vulnerabilities with or without a VDP in place.

A VDP is a way to obtain non-malicious actor contributions (and yes, there is cost to operating a VDP, including handling noise in the signal).

Does the BOD prevent an agency from running a VDP or bounty program that uses vetted reporters? Assuming that an "open reporter" VDP is also provided?

[zmanion]

[Comments 8 and 13]
Determining the complex boundaries of responsibility between third-party providers, partners, non-.gov domains, and the distinction between "used by" and "owned/operated by" is a serious concern, and one faced by many organizations. So to some extent, this problem is not something a VDP is meant to solve. Where are the boundaries for incident response and other security functions?

My interpretation of the BOD is that agencies, including large, complex, and critical agencies like DOE, are directed to incrementally determine their own scope, within the umbrella (and 2 year goal) of "all internet-accessible systems or services." An agency can determine that a third-party system or service is out of scope, primarily because the agency lacks authority to authorize testing on such a system. The BOD points to https://www.justice.gov/criminal-ccips/page/file/983996/download#page=4

[zmanion]

[Comment 14]
To report to CISA: https://www.cisa.gov/coordinated-vulnerability-disclosure-process

To report an ICS, IoT or medical device vulnerability, please email NCCICCUSTOMERSERVICE@hq.dhs.gov or call 1-888-282-0870. When sending sensitive information to the CISA via email, we encourage you to encrypt your messages. Download the CISA ICS public key.

To report an IT Vulnerability, please use the form here: https://www.kb.cert.org/vuls/report/

Is this sufficient guidance?

[cablej]

[1] Given DoD's discovery of 11,000 vulnerabilities through the VDP, it is safe to assume that other agencies have vulnerabilities that could be discovered through a VDP. As a result, this BOD does safeguard systems by protecting against "suspected" vulnerabilities — those that will be discovered by establishing a VDP.

[3] DoD has operated a VDP on all its public facing websites since 2016 without exclusion. It seems odd that DOE systems that "interact directly with DoD" should be out of scope when DoD includes those systems in the scope of its VDP. If DOE is concerned about classified material being exposed via an internet-accessible resource, excluding those systems from a VDP does nothing to prevent the exposures.

[4] The directive make it clear that a VDP is different from a bug bounty, stating a VDP is "similar to, but distinct from, a “bug bounty”". Such clarifications provided in the guidance are helpful in illustrating the differences between such programs and the reasons why an agency may wish to separately launch a bug bounty. It is entirely up to the agency whether or not to launch a bug bounty program (and indeed, as evidenced by DoD's success with Hack the Pentagon, such programs can be valuable supplements to a VDP).

[8] As described by the directive, agencies should determine which components can and cannot be included in their VDP, and should communicate those systems to researchers as such.

[10] The directive allows "the agency to be as transparent as possible about what steps it is taking during the remediation process". Given that the reporter discovered the initial vulnerability, sharing remediation information is helpful to receive feedback from the reporter if a fix is incomplete. That said, the directive does already allow such flexibility in communicating with researchers if necessary.

[11] As a "see something, say something", a VDP must not restrict participation. DoD has received vulnerability reports from around the world and could not have remediated these flaws without keeping its doors open. The VDP template details how agencies can exclude specific types of testing such as denial of service, and that researchers lose safe harbor if they perform those test types. Vulnerabilities exist whether or not we like them; it is the role of the VDP to identify them at every area possible.

[12] The directive states agencies "must not attempt to restrict the reporter’s ability to disclose discovered vulnerabilities to others, with the exception of a request for a reasonably time-limited response period." This makes it clear that there is a timeframe in which reporters cannot share with others, contrary to DOE's statement that "reporters are not precluded from submitting their identified vulnerability to 'others' and can do so at any time". Beyond this timeframe, it is in the interest of all to patch quickly and disclose -- if DOE does choose to "remediate all vulnerabilities immediately" I would consider this a success of the directive in making systems substantially more secure.