-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Enhancement] Secure URL feature #20
Comments
@gamefundas Sure, this is very serious concern. I see two opposite kinds of solution.
I tend to something in between these two solutions and make the restriction an attribute of choice group. Usually plain logs of a group belong to the same root directory (as far as I see in current installations at least). That is why AnaLog already has a special parameter ´plainLogsBaseDir´ which helps to avoid specifying a base dir in every log path. Perhaps we can reuse this parameter as a access restriction. Though it doesn't mean that there is no need to have a global restriction parameter anymore. What do you think about these solutions? |
I guess the flexibility around having global and local base directory makes sense. Currently all our logs are centralized in to a cloud file system. So having that global one will secure things in our use case. |
Ok, I will start with implementing the basic restriction which would deny access to any directory outside of specified one. |
I've researched the subject and come to the following considerations:
Admittedly, the aforementioned considerations can be satisfied by adding the following single parameter into allowedPaths:
- /pub/home/**
- /**/log?/**
- /**/*.log In other words, the restriction can be expressed with YAML array of GLOB patterns, where all elements are considered in OR combination. The array is not required; by default it would have the only element pointing to logs in current user's home directory, i.e. I'm going to implement the feature the way I've described above. @gamefundas, if you have any comments I would be glad to discuss. |
This is well thought through and should work. The user home default is nice idea that way nobody is exposed to begin with. In case if we have a /pub/home/** then the url access will only allow url’s that match http://host:port/pub/home/**some.log? |
@gamefundas Yes. |
Actually I wasn't aware of the remote feature, this sounds new. Its already released or work in progress. Couldn't find any documentation on the agent. |
I see it now Would like to understand this architecture better, are we saying that Analog can be configured to run as a server and applications/nodes can run Analog agent to expose their logs. The only trouble I see with this approach is maintaining application.yaml. As applications run on ephemeral infrastructure defining or managing a static configuration of nodes is going to be a daunting task. Formulating a URL scheme that translates the path to actual host/target can help move away from the static definition. An example for remote log fetch would be something like target - localhost, hostname:port, ipv4 address (localhost will serve logs from current host or server) - Default agent port 7801 if not specified Support for something like this, may be in addition to cluster configuration (which makes sense for complex and detailed setup) can help source remote logs without any declarative configurations. Basically this will allow sourcing logs from any Analog agent without any configuration. Just a high level thought. |
This is a released feature. It was initially implemented in v0.7 as a part of composite log support. Then, in v0.11 it was exposed as a standalone feature so that from this version on you can refer to logs residing on remote servers by specifying
You can also look at choices.yaml sample file to find examples of remote logs addresses, e.g.: - path: node://backupNode/home/upc/node-app.log # 'backupNode' is the name of node as declared in 'nodes' config section
TL;DR - Yes, that's right (with some notes).
Yes, that's the problem. The solution you propose looks good but I doubt about its portability because inclusion of actual nodes addresses into AnaLog pseudo URLs (log paths) would make those URLs un-reusable i.e. every time a node changes its address all the URLs become invalid. It may not only disturb end users with periodical 404 error (or something like this) but also would make logs paths unsuitable for storing in browsers bookmarks (which is a quite handy feature now). That is why I'd like to propose another solution (which is actually already drafted in current versions of AnaLog). Instead of declaring explicit addresses anywhere (be it an URL or
@gamefundas, what do you think about such an approach? |
I am thrilled hearing service discovery as originally my thinking was around it but then at the same time restrained myself as it could complicate Analog by integrating in to other products. We use Consul extensively for service discovery so this could be a blessing in disguise. However please note that, this may not work in cases where processes are independent and not discover-able in nature. Still your idea would address a high percentage of use cases and seems to be the right direction. The one other option which I can think of, which can avoid deeper integration with other tools is, for the Analog agent to register to the server with a list of log remote locations that's available to browse. Users have full control over their agents and its configuration so essentially they will always configure it in a way that seems fit. The server is simply using the registered information to create url's/path to source the remote file. This will also help populate the drop down with remote locations. Not sure if this is exactly what you have now and I misunderstood the example. |
Yes, I totally agree with you and that is why I've mentioned early the ability to specify nodes configuration statically. This would serve as a fallback for any environment with no service discovery installed or for those who use AnaLog in a simple deployment scenario. Also it can appear not so easy to support multiple service discovery implementations at the same time but I hope Spring Cloud would help with it. Speaking of your proposal I can say it is quite similar to what we have now. The main difference is that currently agents do not tell the servers about their available locations. Instead, server populates the drop down with data from it its own static configuration ( |
I feel strongly that the agents defining the log locations would guide the design better as agents by all means know their logs better and are the ones that change state often. Whenever applications deploy they would restart their agents and new configurations will come to effect. The server will not have to be touched or reconfigured in any way. It can be configured only for its own logs. I have 90 applications operating in my VPC which typically register to central infrastructure such as metrics, discovery, syslog monitoring etc and these apps are constantly deploying/redeploying and the numbers continue to shrink and grow depending on autoscaling loads etc. In such a dynamic environment with responsibilities divided between various team, I don't have to ever worry about whats on the central log viewer (Analog) as log locations are subscribed by Analog server whenever the agent connects. Else I will be constantly in the process of tweaking adding/removing editing configurations in production and then restarting Analog to be able to support new or moved log locations. Worth considering this feature for sure. |
Ah, it seems I misunderstood you, sorry. Let's try to clarify it. What I understood early (from that comment)AnaLog should use an external service registry (like Consul) to store the information about current agents locations. Additionally, agents can register themselves to some AnaLog instance (a server) in order to populate the server's drop down list of log choices. What I see now (from this comment)AnaLog should not use an external service register. Instead, one of AnaLog instances would be elected as a central one ("a server") and then all other instances (agents) would register themselves to this server providing it with lists of available logs to watch. In other words, we don't refuse from having a service registry, rather we're making it an internal part of AnaLog cluster. If this is correct notion, I'll try to describe it a little bit further.Since we don't have to support any external service registry anymore, we're free to choose any registry that best suites our needs as internal one. Netflix Eureka seems a good choice because it's shipped as a library and thus can be easily integrated into AnaLog instance making it a full-fledged service registry for the agents (I have an experience of building such applications). At the same time Eureka protocol supports meta data which can be useful for us to propagate agents' log locations to the server. Furthermore, as Eureka clients (in every AnaLog agent) performs periodical queries (heartbeats) to Eureka server (AnaLog server), we can use it to dynamically refresh the content for drop down list on the server. Though it should be stressed that such choice configuration model should be an additional option rather than a requirement. The reason is that there is an AnaLog deployment (at least one I know) where all 7-9 instances are controlled by single administrator. He doesn't store any log choice configuration on agents because it would make him control 7-9 files on 7-9 servers. Instead, he collected all the configuration on the (single) server and controls only it. Looks reasonable to me. P.S. Just in case, I'd like to remind that every AnaLog instance can play both roles: server and agent. So that any person or a team responsible for some AnaLog agent can work with its logs without any additional server, just by opening agent's web UI. |
Yea, I got a little excited about service discovery but then soon realized it would mean adding dependency on other frameworks which is not a good thing. But yea this can be left to users choice. Also this depends on team to team. In our case the DevOps or infrastructure owners are a separate group who provide centralized services such as logging, security, monitoring, provisioning, database change management etc. The application owners are building business services and merely interacting or configuring their apps to use the central services. All configuration aspects (almost) are provided for the app teams to figure out how to customize it to their needs. Currently based on cloud configuration, all application logs are streamed to a central log server (logstash running along with a primitive log ui) which makes it available for viewing. What form, location, hierarchy etc the logs should be made available in the UI is fully controlled by the log agents (Filebeat) using simple metadata. This gives full control to the application teams as they know best, of the various logs (components) they want to publish. That said will leave this to you to decide. We are perhaps in agreement for the most part. My second part of the solution was mainly to get rid of the log shipping aspect completely (No Filebeat and Logstash in my case). As it also makes AnaLog as an end to end solution for viewing logs with zero configuration (no knowledge about client log locations) on the server side. |
Yes, now I see your use case and seem to understand how to fully address it in AnaLog. Thank you for the description! The only thing I'd like to clarify a bit more is an aspect of storage. Do you currently use any persistence under Logstash (e.g. Elastic like in ELK)? As you know, AnaLog doesn't provide any storage for logs and is not meant for because AnaLog is a log viewer. But when we compare it with a persistent solution like ELK we should keep in mind this significant difference. Just a reminder. |
Our current pipeline is App -> Cloudwatch -> Lamdba -> Redis -> Logstash (fanout to multiple outputs) => -> Filesystem (EFS) -> Custom Log Viewer Yes we have persistence in Amazon EFS (central file system) and Elasticsearch |
Ok, thank you for clarifying. |
Awesome thanks for the update. Look forward to the release. |
2. Fixed context tests 3. Added JavaDoc for the newly created classes
I've drafted a document with description of the feature. |
Log Access ControlOverviewAnaLog can restrict access to certain logs to prevent sensitive data from leaking (e.g. queries like ℹ️ As of v0.12 the restriction is provided for file logs only (both remote and local ones). To manage access restrictions the administrator can use AnaLog ships with the following default access configuration: allowed-log-locations:
file:
include:
- ${user.home}/**/*.log
- ${user.dir}/**.log
exclude:
- ${user.home}/**/.**
symlink-resolution-limit: 1 It allows reading of non-hidden logs from current user’s home directory tree as well as reading any logs from AnaLog’s own working directory tree. If you want to understand the access restriction mechanism in depth, please read the following section. DetailsAnaLog performs access check in two cases:
The checks are performed on the current instance’s agent side, i.e. every AnaLog instance controls its own logs only and doesn’t restrict others in any way. As a result, there is currently no way for the administrator to declare all the access configuration on single instance (“master”) and make it come to effect everywhere. The access control procedure consists of the following steps. 1. Checking if file log access is denied at allAnaLog uses restrictive access model, i.e. everything that is not explicitly allowed, is denied. Therefore there must be at least one entry in
This can be useful for AnaLog instances that are not intended for working with file logs (e.g. if they are meant to work with container logs only). 2. Normalizing the pathOn this step AnaLog:
These three steps provide subsequent checking with actual log paths rather than various pointers like relative paths, symlinks, etc. 3. Checking the path against including patternsOn this step AnaLog checks normalized path against ℹ️ Simply put, Glob patterns lets you specify a path either as usual e.g. The requested log path must match at least one of the patterns to pass the check. AnaLog processes the list from top to bottom, the first matching pattern wins, no other patterns are examined in this case. If no patterns are specified, an error is returned immediately (see first check above). When no matching patterns are found, the log is denied from reading and the following message is returned to the client:
Note that matching to any of including patterns is not sufficient for a log to be allowed for reading because AnaLog also checks every log path against excluding patterns (see next). 4. Checking the path against excluding patternsOn this step AnaLog checks given path against The requested log path must not match any of the patterns to be allowed for reading. AnaLog processes the list from top to bottom, the first matching pattern wins, no other patterns are examined in this case. If no excluding patterns are specified, the path is considered allowed for reading. When matching pattern is found, the log is denied for reading and the following message is returned to the client:
Excluding patterns are meant to somehow narrow the including patterns. They also can serve as exceptions from the implicit list of logs matching the including patterns. To understand it better, you can formulate the access control logic as the following sentence:
Note that including patterns (<inclusion>) are combined with or while excluding patterns (<exclusion>) are combined with and. Variables substitutionYou can refer to various external values from within access settings with the help of
For example: allowed-log-locations:
file:
include:
- ${user.home}/**.log # JVM built-in property (current OS user home dir)
- ${JAVA_HOME}/lib/classlist # OS environmental variable
- **/${nodes.this.name} # locally defined parameter (AnaLog's current node name)
exclude:
- **/${user.name}.log # JVM built-in property (current OS user name) |
During the testing its appeared that some tail implementations (at least GNU Core Utils which is enough) do follow symbolic links unconditionally. It can be used to bypass the checking AnaLog performs at the start of a new tailing process. To address the issue I've added additional handling for certain tail events and now AnaLog is capable of detecting path manipulations at watching runtime as well. It took some time to implement. |
By now, the feature seems completed and I'm going to close the issue soon. @gamefundas, I still would like you to review the description of the feature before I make it generally available. I've updated this document and placed it to Wiki for your convenience. |
Log Access Restriction Feature (#20)
Apologies, was away a little bit on a new engagement and couldn't attend to our Dev Ops track. I read through your notes on the overall design around folders and its very comprehensive. Love the flexibility overall. I will try this in our new log pipeline stack to see how it operates in a real world scenario. Planning for a release soon? |
@gamefundas Yes. Currently I'm working on translation (#22) which doesn't seem a big deal. After that the release will be published. I will notify you when ready. |
I noticed the following warning that reminds about the security exposure around allowing access for any log file using URL feature
https://github.com/Toparvion/analog#heavy_exclamation_mark-security-caution
I think this can be very concerning for many who would use this tool. A simple configuration option in case of URL feature could be that if the application has a property like "url_log_base=/apps/logs" then everything on the URL can be in relative context to the base directory. This will restrict people from seeing anything outside of the defined scope. URL hacks like "../../" may need to be thought through but this is still better than what we have currently.
If this needs to be more flexible then it can support a list of such base directories. Otherwise in my case the info sec guidelines wont allow me to deploy this tool. Also think Java permissions can be leveraged but then all of those would require more configurations to deal with.
The text was updated successfully, but these errors were encountered: